1. Field of the Invention
The present invention relates to a novel consistency management method that applies redundancy encoding and decoding of data blocks across a plurality of interconnected data stores in a network.
2. Related Art
Erasure coding is an error correction encoding and decoding scheme. Erasure coding is applied to a set of data stores to generate one or more redundant data blocks to protect against erasure of the actual data. A data store refers to a persistent memory for a given data block. In the event of data loss, part of the remaining original data blocks and part of the redundant data blocks can be used to recover the entire original data set. In the event of a device failure (e.g., a data store failure), and when a replacement device is available, the recovered data blocks can be used to reconstruct a latest consistent state of the failed device for distribution to the replacement device.
There are many different types of erasure or error correction coding known in the art. These include, without limitation, data mirroring, parity coding, and algebraic-based coding. Data mirroring and parity coding generally create one additional data block from a number N of original data blocks. This type of coding scheme allows a single data set to survive through one failure while still having the capability to reconstruct the original data. Multi-dimensional parity coding may be applied across several data sets to allow for two or more concurrent failures. Such multiple dimensional parity coding supports multiple failures by combining multiple encoded data sets. Thus, for example, in the case of the two-dimensional parity coding, vertical and horizontal data sets individually allow only one failure, whereas the combination of both data sets allows for two failures. Algebraic-based coding schemes, such as a Reed Solomon code, take N data blocks and generate N+M data blocks. This well-known process is illustrated in FIG. 1, using the N data blocks 101, and an encoding algorithm 103, which generates a (N+M) data set 110 having M checksum blocks 105. The associated recovery procedure also is well-known as illustrated by example in FIG. 2. In this example, blocks D2, D4 and C3 are presumed to be bad; nevertheless, the decoding algorithm 202 can still be used to recover D2 and D4, which blocks are then used to recover the checksum block C3.
In general, when a data failure occurs, this type of algebraic-encoding scheme requires only any random N copies of data blocks from the N+M number of data blocks to reconstruct the lost data. Thus, algebraic encoding supports up to M concurrent failures in a single data set. To apply algebraic-based coding, when an encoding process detects a data change from one data store, it must generate and update all M redundant data blocks. In other words, it is required that the process have the capability to ensure all M redundant data blocks are completely updated. Because the process may fail during the update (during which other failures may also occur simultaneously), there needs to be a self-healing technique to recover the data from the failure(s).
When applying multiple redundancy erasure coding (such as algebraic-based coding) to data blocks in a set of data stores, one also needs to consider the consistency of the entire data set as well as the correctness of the data blocks. A set of data blocks is considered to be consistent if all the redundant blocks are generated from all the original data blocks. For example, in FIG. 1 all the redundant data blocks 105 are generated by all the data blocks 101 using the encoding algorithm 103. In the event of an update failure during encoding, the set of data blocks may become inconsistent, as indicated in FIG. 3 and as described in more detail below. The common solutions to address an inconsistent set of data blocks are: do nothing, manage the situation on a case-by-case basis using a specific application, or simply re-execute the encoding process to regenerate all the redundant data blocks in a data store, as illustrated in FIG. 4. The problem with the first solution is that the set of data blocks becomes inconsistent and incorrect. If failure occurs to the inconsistent data set (as shown in FIG. 3), the decoding process would generate incorrect data. The second solution may result in implementation complexity. The main issue with the third solution is that it does not address data correctness. In particular, when a procedure that modifies a data block fails in the middle of the operation, the data within the block is inconsistent and incorrect, as illustrated in FIG. 5. The third solution also does not address the situation where there are data store failures. When a data store failure occurs, and if the data store is used for storing the actual data, the re-encoding process cannot proceed. If the data store is used for storing a redundant block, then the re-encoding process is not able to update all the redundant data stores, in which case there is no way to identify which data block is inconsistent when the unavailable data store becomes available again.
While multiple redundancy erasure coding could increase data reliability, it has not been possible to apply it to persistent data stores that are being constantly updated. To address this deficiency, there needs to be an efficient and simple consistency management method in an encoding process to apply the erasure coding. Such a consistency management method would allow data stores to self-heal from failures, and it would ensure data consistency and correctness among all the data blocks.
The present invention addresses this need in the art.